Mathematics Achievement in the Netherlands and Appropriateness of the TIMSS Mathematics Test

Dutch students performed relatively well on the TIMSS mathematics test for population 2, although influential mathematics educators heavily criticised the appropriateness and fairness of the test in terms of the new, applicationand inquiry-oriented mathematics curriculum which was implemented beginning in August 1993. This new intended curriculum differs drastically from the previous more formal and abstract curriculum from before 1993. For both policy makers and mathematics and science educators in the Netherlands important questions are the following: how should this discrepancy between students’ performance and curricular appropriateness of the test be interpreted, and is there a discrepancy? Via expert appraisal, data have been gathered about the extent to which the TIMSS mathematics items are appropriate for this new intended mathematics curriculum. In addition, teachers were asked to judge the appropriateness of a selection of mathematics items in terms of the implemented curriculum. In this article the outcomes of the analyses with regard to the overlap between, on the one hand, the test and, on the other hand, both the intended curriculum and the implemented curriculum are described. In addition, as far as the intended curriculum is concerned, test-curriculum overlap outcomes are related to students’ performances on the test. INTRODUCTION AND RESEARCH QUESTIONS The component with the highest and at the same time most challenging profile in any comparative study such as TIMSS is the achievement survey. All participating countries wish to ensure that the achievement items Address correspondence to: Wilmad Kuiper, University of Twente, Faculty of Educational Science and Technology, Department of Curriculum, P.O. Box 217, 7500 AE Enschede, the Netherlands. Tel.: +31 53 4893757. Fax: +31 53 4893759. E-mail: kuiper@ edte.utwente.nl Manuscript submitted: March 30, 1998 Accepted for publication: January 19, 1999 D ow nl oa de d by [ U ni ve rs ite it T w en te ] at 0 1: 58 0 4 D ec em be r 20 14 WILMAD KUIPER ET AL. 86 used in the survey are appropriate for their students and reflect their mathematics and science curricula, enabling their students to give a good account of their knowledge and their ability and ensuring that international comparisons of student achievement will be based on a ‘level playing field’ insofar as possible (Garden, 1996; Garden & Orpwood, 1996). In TIMSS, like in all IEA studies, an achievement test that matches the content of the mathematics and science curricula of all participating countries was an unattainable goal though. Instead, as a compromise, the development process aimed at a test that was ‘equally unfair’ to all participants. Due to the number of countries (more than 40), the task of putting together the achievement test for the three TIMSS populations was immense, and took more than 3 years to complete. As is customary in IEA studies, developing the item pools was a co-operative venture involving all countries (including the Netherlands) during the entire process. From its initial stage of development though, influential mathematics educators in the Netherlands heavily criticised the TIMSS mathematics test for population 2. To their opinion, the test was inappropriate and unfair in terms of the new, applicationand inquiry-oriented mathematics curriculum that was implemented beginning in August 1993. Within this context and against the background of the three well-known conceptual levels of curriculum (cf. Editorial in this special issue; see also Beaton, Martin, et al., 1996; Beaton, Mullis, et al., 1996; Robitaille & Maxwell, 1996), the research questions that are addressed in this article are: (i) How appropriate is the TIMSS mathematics test for population 2 in terms of both the intended and the implemented mathematics curriculum in the Netherlands? (ii) How do Dutch students perform on the TIMSS mathematics test for population 2 and how should one interpret these students’ performances against the background of findings with regard to the curricular appropriateness of the test? SAMPLE AND RESEARCH GROUP In the Netherlands – as in all of the other northern-hemisphere countries – the TIMSS data collection took place in spring 1995. A two-stage sample design was used. The first stage involved the selection of schools. Next, within each participating school random procedures were used to select one intact class at the upper target grade and one at the lower D ow nl oa de d by [ U ni ve rs ite it T w en te ] at 0 1: 58 0 4 D ec em be r 20 14 MATHEMATICS ACHIEVEMENT 87 target grade. All the students in those two classes participated in the TIMSS testing (same test for both classes). As part of the second stage the mathematics teachers from the classes involved were selected. In the Netherlands, this approach yielded a stratified representative sample of 95 schools1, with 2,160 students from secondary 2 (upper target grade) and 2,027 students from secondary 1 (lower target grade) participating in the testing and the filling out of a student background questionnaire. Of the classes tested 155 mathematics teachers filled out a teacher questionnaire. In the Netherlands, secondary education caters to students aged 12 to 17 or 18 years old. Students may follow one of four main ability tracks (Kuiper & Knuver, 1997): – Junior secondary vocational or prevocational education, known as VBO. This is a 4-year program, specialising in technical, home economics, commercial, trade or agricultural studies. – Junior general secondary education, known as MAVO, a 4-year long program. – Senior general secondary education, known as HAVO. This is a 5-year program, preparing students for higher vocational education. – Pre-university education, known as VWO. It is a 6-year course, preparing students for university and higher vocational education. Secondary schools offer various combinations of these tracks. In the sample a distinction has been made in six combinations of tracks. However, at classroom level outcomes are reported for two main groups of students: students in VBO/MAVO and students in HAVO/VWO, both in secondary 1 and 2 (cf. Bos, Kuiper, & Plomp, in preparation). 1. A sample of 95 schools means a school participation rate of 63% after replacement schools were included. Although such a response rate is not uncommon in the Netherlands, it is below the TIMSS sampling participation standard of 85%. Therefore, in the international tables the Netherlands appears as a “country not satisfying guidelines for sample response rates”. From an analysis of the quality of the Dutch sample, though, it has been made plausible that the schools that participated in the study were representative for the population. The analysis consisted of a comparison between the schools in the sample and the schools that refused to participate with regard to average school leaving exam scores for mathematics. From a one way analysis of variance per school type it appeared that there were no significant differences in average scores between the two groups of schools. D ow nl oa de d by [ U ni ve rs ite it T w en te ] at 0 1: 58 0 4 D ec em be r 20 14 WILMAD KUIPER ET AL. 88